1. Field of the Invention
The present invention calibrates the orientation of an orientation sensor mounted on an image-pickup device with reference to the image-pickup device in order to measure the orientation of the image-pickup device.
2. Description of the Related Art
In recent years, mixed reality (MR) for coupling real space to virtual space and eliminating the joint therebetween has been actively studied. For example, a known image-display device for presenting the MR is formed, as a device for producing a display image generated by superposing the image of virtual space generated according to the position and orientation of the image-pickup device (a virtual object, text information, and so forth that are drawn by computer graphics or the like) on the image of real space that is picked up by an image-pickup device such as a video camera.
The image-display device may have applicability in navigation for overlaying the name and/or guide information of a famous building or the like on an image obtained by photographing a built-up area on the real-space as a virtual-space image. Further, the image-display device may also have applicability in view simulation for superposing a computer-graphics image of a building to be constructed on an image obtained by photographing the construction site of the building.
A common problem occurs regarding how precisely registration between the real space and the virtual space can be attained in a case in which the image-display device is applied to the navigation and a case in which the image-display device is applied to the view simulation. Therefore, considerable research has been done for solving the problem. For performing the registration between the real space and the virtual space, camera parameters (intrinsic parameters and extrinsic parameters) for generating an image of the virtual space are synchronized with camera parameters of the image-pickup device at all times. Where intrinsic parameters of the image-pickup device are known, the registration problem of the MR is attributed to the problem of extrinsic parameters of the image-pickup device, that is, the acquisition of the position and orientation of the image-pickup device in a world-coordinate system that is set in the real space.
T. Hollerer, S. Feiner, and J. Pavlik, Situated documentaries: embedding multimedia presentations in the real world, Proc. International Symposium on Wearable Computers '99, pp. 79 to 86, 1999 discloses a method for obtaining the position and orientation of an image-pickup device by measuring the orientation of the image-pickup device by using an orientation sensor and measuring the position of the image-pickup device by using a global-positioning system or the like, as the method for obtaining the position and orientation of the image-pickup device in the world-coordinate system that is set in the real space.
TISS-5-40 of TOKIMEC INC. and InertiaCube2 of InterSense of the U.S. are used, as representative examples of the orientation sensor used for the above-described method. Each of the above-described orientation sensors mainly includes three gyrosensors for detecting an angular speed in each of three-axis directions and three acceleration sensors for detecting an acceleration in each of the three-axis directions, and measures three-axis orientations including an azimuth angle, a pitch angle, and a roll angle by using the angular-rate measured values and the acceleration measured values in combination. Usually, the angular information obtained only by the gyrosensor only shows a relative change in the orientation with respect to the orientation at a predetermined time. However, each of the above-described orientation sensors can calculate absolute angles of inclination, where the angles of inclination include the pitch angle and the roll angle, with reference to the gravitational direction by measuring the Earth's gravitational direction by using the acceleration sensor.
The orientation-measured value transmitted from the orientation sensor represents the orientation thereof in a sensor-coordinate system that is defined by the orientation sensor independently of the world-coordinate system. In the case of the above-described TISS-5-40, for example, the sensor-coordinate system is defined, where the gravitational direction (downward direction) is determined to be the Z-axis, and the orientation-sensor front direction in an X-Y plane defined by the Z-axis at the time when the orientation sensor is initialized, is defined as the X-axis. Further, in the case of the above-described InertialCube2, the sensor-coordinate system is defined, where the gravitational direction (downward direction) is determined to be the Z-axis, and the north direction in an X-Y plane defined by the Z-axis that is pointed by an internal geomagnetic sensor, at the time where the orientation sensor is initialized, is defined as the X-axis. Thus, it is usual that the orientation-measured values measured by the orientation sensor do not represent the orientation of an object to be measured (the image-pickup device in the case where the image-display device for presenting the MR is used) in the world-coordinate system, that is, information to be acquired.
That is to say, since the orientation-measured value obtained by the orientation sensor cannot be used in its original format, as a value representing the orientation of the object to be measured in the world-coordinate system, coordinate transformation of some kind may have to be performed. More specifically, coordinate transformation for converting the orientation of the orientation sensor into that of the object to be measured (hereinafter referred to as local-transform) and coordinate transformation for converting the orientation of the orientation sensor in the sensor-coordinate system into that of the orientation sensor in the world-coordinate system (hereinafter referred to as world-transform) may have to be performed.
The world-transform is defined by the orientation of the sensor-coordinate system with reference to the world-coordinate system. Moreover, the local-transform is defined by the orientation of the orientation sensor with reference to the coordinate system defined by the image-pickup device.
As has been described, the sensor-coordinate system is determined according to the gravitational direction. Therefore, in which direction the gravitation axis of the sensor-coordinate system (the Z-axis in the cases of TISS-5-40 and InertiaCube2) points in the world-coordinate system can be uniquely determined according to the definition of the gravitational direction in each of the sensor-coordinate system and the world-coordinate system. By using the above-described information, the world-transform can be obtained, where the rotation angle around the gravitation axis, i.e. azimuth, is kept in an unstable state. More specifically, a three-dimensional vector 1 showing a vertically-upward direction in the world-coordinate system and a three-dimensional vector g showing a vertically-upward direction in the sensor-coordinate system are determined and an angle β which the three-dimensional vector 1 forms with the three-dimensional vector g is calculated based on the inner product g·1 of the three-dimensional vector g and the three-dimensional vector 1. Further, a normal vector n=g×1 of a plane formed by both the vectors is calculated based on the outer product of the three-dimensional vector g and the three-dimensional vector 1. Where a rotation matrix R*WC for performing coordinate transformation, where vector n is determined to be the rotation axis and the angle β is determined to be the rotation angle, is calculated, the rotation matrix R*WC denotes the orientation of the axis corresponding to the gravitational direction with reference to the world-coordinate system.
The applicant of the present invention proposed the above-described method for calculating the world-transform in U.S. Patent Application Publication No. 20050068293 filed Sep. 30, 2004.
In the case of the world-transform, only a rotation angle around the gravitation axis, i.e. azimuth, is unknown.
The above-described unknown parameter is geometrically equivalent to an “azimuth-drift error” or an accumulated error of the orientation-measured value around the gravitation axis. The azimuth-drift error occurs with the passage of time, where the orientation sensor is the gyrosensor. Therefore, when the rotation angle around the gravitation axis in world-transform is considered, as the “initial value of the azimuth-drift error”, the unknown parameter can be considered, as part of the azimuth-drift error of the gyrosensor, where the azimuth-drift error changes dynamically. As a result, the world-transform can be considered, as a known value that can be derived based on the relationship between the gravitation axes. Further, automatic measurement (automatic correction) of the azimuth-drift error can be performed according to known methods using image information, so long as the local-transform is known.
However, no methods have been proposed for easily and correctly deducing the local-transform wanted to be known. Hereinafter, the task for deducing the local-transform of the orientation sensor is referred to as “sensor calibration”, or simply “calibration”.
For example, according to one of calibration methods that have been used, an operator adjusts each of the values by trial and error until correct registration is achieved by increasing and/or decreasing the values of parameters representing the local-transform in an interactive manner through an input unit of some kind. However, the above-described method is difficult to perform. Further, according to another method of the calibration methods that have been used, the image of a virtual object generated based on a predetermined orientation is produced and shown on an image-display device and the operator adjusts the orientation of the image-pickup device by visually checking the orientation so that the positional relationship between the shown virtual object and the real space becomes correct. Then, the above-described two coordinate transformations are calculated based on an output currently transmitted from the orientation sensor. The above-described method is disclosed in U.S. Patent Application Publication No. 20030080979, filed Oct. 21, 2002. Although this method is easier to perform than the method where the parameter values are adjusted in the interactive manner, it is dependent on the visual check of the operator. Therefore, this method is susceptible to improvement in correctness and reproducibility.